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INFORMED. DECISION MAKING , 
j (Evaluation You Can Use) 

M. C. Linn and W. M. Laetsch 
' Lawrence, Hall of Science 
• * University of California V 

^ Berkeley; California 94720 . 

Evaluation has gained a rather unenviable reputation; ,Just menti6n the 
word and defenses rise to the occasion. Some negativism undoubtedly stems 
from a feeling that evaluation will be destructive and discouraging. One 
oiten hear-s, "It's easier to criticize than! 1 create". Many program developers 
explain that they 'already have a large supply of criticism, "Just attend a 
staff meet ij\g and see for yourself how critical everyone is!" A very conmon 
note is: "We (Jon't have time (or money) for evaluation". As a result, 

evaluation is a small and often non-existent element, in programs of insti- 

* • „■*■».• 

tutions concerned with informal education. 

When evaluation is carried out, it's usually of exhibits that are 

cqtipleted and cannot be changed. Evaluation of the Effect of a completed 

exhibit is pften called simulative evaluation* While suwnative evaluation 

has value, it is of no use in improving an gxhibit. At Lawr^&e Hall of 

Science we have been developing a model fl>r evaluation which results in 

improvemeht of exhibits and programs during their development . This type 

of evaluation is often called formative. In, this paper we will argue that 

museums currently have a. greater need for formative than for suhmative / 

evaluation, (and we will ^suggest which areas of formative evaluation are 

of primary concern. . , *^ 



The major ta^in formative evali^a.ti^n is to ^ect questions that 
need t^o be ansWered and to develop methods to answer these questions. x 
Museun evaluation should be used to improve the programs, exhibits,' and 
products so that they serve the target population better. The users of 
evaluation information are progranl and exhibit planners -and developers . 
They need information which will increase the likelihood that they will 
make effective decisions. The eva Am tor will be most effective when ' ' 
working in the program from the early planning stages as a member of the 
exhibit development team. ^ 

The most well known evaluation studies have been done by educational 

psychologists concerned with evaluation of curricula in schools, industry, 

the military or other institutions concerned with formal education. The' 
* i 

major question in well known studies has been summative: whether users of 
a particular program- learn more than users of another program.. These 
educational horseraces have frequently been inconclusive for many reasons 
including: ' • 

1) the difficulty offsetting up truly controlled experiments in 
the real world of; schools, and ' 

2) the difficulty of comparing programs which have different goals 
(fr-g. "to have children grow plants" vs.* "show children that 
plants grow better when they have plant food") . 

Normative evaluation is, of course, used in development of many 

" ^ 

educational programs, but it is not so well publicized as sunmative eval- 
uation. After all, educational consumers making decisions- about which 
program to use are' likely 'to be influenced by summative evaluation. On the 
other hand, the prograuft will be greatly enhanced by fonnative^evaluation* 



' • " EV/ J^TION 1^ TIE MUSEUM OR SCIENCE ^NTER 

i ■ 

u 

It is tempting jt'o focus evaluation In museums^ and s science and tech- 
nology centers on what users learn. Most currently available studies of 
museuns have been sunmative (Lakoda 1975; Screven, 1975), although Shettle 
(1975) has reported on a very useful formative stxhd^ \< 

While educational goals are important , it is imperative that the 
museums focus on*, exhibit characteristics and user interests as well„ 
Museums are concerned Kit h how to bring visitors back while, schools' have 
a captive audience. What we'need to know at the beginning is How to , 
attract antf how to interest visitors. Only after we have the visitors 1 
intention can we hope to teach than spmething^T Ntuseum visitors are not 

9 

obliged to comd, usually stay for at most two hours, and can choose what 
they want to do! 

Formative evaluation also makes sense Development sta£f for * 

»■ 

successful public programs such as Disneyland are \fery responsive to 
visitor* feedback.* Public schools, of course, httVfe not been as directly * 
^concerned wi^h immediate user satisfaction for mafiy reasons including 
the need to impart knowledge. With the^current decline in the school 
population, and dearth of educational funds, schools may well increase - 

their emphasis on user satisfaction. ' The school model rather than the 

i 

Disneyland model appears to have^inf:luenped museums and science, centers. 
For instance, most museum classes take place in classrooms with a great 

deal of teacher talk. Like school, museum programs are frequently' 

» •' / h 

^difficult for' the users to under stan^-^nd. liberally supplied with "facts"; 

' • . 'S 

user enjoyment is not high priority. . . - -\ * 
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We could all learn from the Disneyland model. In developing science 
center programs, there are all, sorts of terrific opportunities to make choices ; 
The amount of* visitor participation, type of graphics,, exhibit grouping, and 

many other activity characteristics arg^ flexible. Evaluations can be used to 

■* 

help the center staff make these decisions to attract and keep visitors. , . 

Why have museums avoided formative evaluation? MajorYeasons are time 

* * * 

and money. Many programs never reach the revision stage so there is no 

opportunity to use information from, evaluation. There is also. the curator 

syndromic exhibits are designed to appeal to other curators, not the 

museii«L-yisitors. Naturally these exhibits are incomprehensible to the 

general public. , Th^ere is also a feeling that museums know what jysers need, 

\ 

i < . 

even though it's not what they want. Again schools have traditionally been * 

viewed as places which provide what is needed - arid somehow it's better 

if it's a bit unpleasant to get what is needed. The "take it or leave it" 

attitude is hardly appropriate when serving voluntary -visitors to museums. 

INFORMED DECISION MAKING 

i 

Our 'model of evaluation is called "Informed Decision Making". /In 

I 

this system, the evalyator first helps to create choices, and then gathers * 
evidence so that a reasonable decision can be made. * 

Creating Choices '* "■ . ! . 

The evaluator can help exhibit developers by identifying choices that 
they are making or that could* be made. One area where choides might be 
created is the, learning environment. Possible envirbjnments range from 
traditional classrooms to total entertainment. 0y installing^ predominantly 

*\ 

♦ / 
** " • . . 

graphic j^xJtiibit, the exhibit developer has chosen not to install a participatory 

■ ' 7 
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exhibit. . The evaluator, involved in early exhibit planning, might create 
choices by interviewing a few visitors a^out their preference^ for graphics 
Sometimes the invitation to consider other ways to handle a particular 
situa*i4$n is useful. Just thinking about new possibilities can create 
choices. * ^ 

Another tool used by evaluators is to relate ^research findings in 
psychology rind education to the Quest ions faced by exhibit developers QSee, 
for example, Thier and Linn, 19^6) . 

One technique which is particularly useful for creating choices in the 
museum environment is systematic observation. At LHS we have used photographs 
to get an idea about how visitors respond to our environjtent . '„ (See Figures * 
1-4). Our visitors come in all sizes, shapes, and ages, ajid some of them are 
not stimulated' by our program. Failure to serve particular group constitutes 
a choice, and we should be aware of it. . It is a common observation in science 

4 • 

centers that younger children love buttons. Unfortunately, we rarel^capita- 
lize on this fascination. We observe that our many family groups engage in 
more verbal interaction than do^ same age visitor groups. These observations 
are but a' few of those which can aid us in understanding the choices we make 
when we install exhibits. 

C 

« 

f . 

Making Choices 

'- « 
the second function of the evaluator assisting in Informed Decision 

Making is to gather evidence to help exhibit planners to decide between the 

9 

choices that have been created. Feedback is needed to make reasonable • 
decisions. The techniques of curriculum' evaluation, education, and ..psycho- 
logy can be relevant in this process. The most common approach is visitor 
interviews. Choosing the right questions is actually not easy, but can be 
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learned through exp^lence, * 

w * 

** 

This evaluation model is illustrated in the development of a parti-" 
cipatory Astronomy Exhibit at LHS. This recently opened exhibit, developed 
by Alan Friedman and LHS s£aff , funded in part by the' National Science 
Foundation, and a gift from Milium K. Holt, has/been in development for 

i „ / 

* v 

almost two years. Description of tfie advantages otf" formative evaluation 
for the development of this exhibit is described in the accompanying article: 
"Star Games: The Pain$ and Pleasures of Formative Evaluation" (Easdfo and 
Friedman). * " v 

As you might have anticipated a,fter the development "phase, once Ae 

astronomy exhibit was ope*h to the public, the devejjtopgrs wanted to know if 

/ * 

people learned while doing the [activities. This question is still in the 
area of formative evaluation e^'ecially since it- is possible to change many 
aspects of the exhibits. 



There are various ways to measure exhibit effectiveness: 

. - ; ■ * c 

1) A common method to assess exhibit effectiveness is to measure 
popularity. This is far from a simple question. It can be 
' measured in terms of numher. of' users, number of return visits, 
likelihood of being remembered or many other wa*ys. Our obser j 
\ ' vat ions^ indicate that popularity should also be. evaluated in 

terms of who the ijsers^ are., > We have found that some exhibits « 
are popular with one group, and actively disliked by another. 

i 2) Another way to measure exhibit effectiveness is the length of 
._.y time visitors spend at the exhibit. Clfearly it would be 

impossible to, learn from an exhibit if you didn't visit it, . 
and probably the learning potential for an exhibit is closely 
related to the time spent at the exhibit. Observation at LHS 
indicates that users spend more time on exhibits that have a 
greater participatory potential. For this reason alone, parti-, 
cipatmy exhibits are likeiy^to result in greater fearriing 
. than non-participatory exhibits. ' ; 



3) 



A third mfttod of measuring effect ivenesfs^^ 
generates ^interest in science. In, the case / of the astronomy, 
exhibit, we asked LH$ school group visitors, to indicate which 
of 3 posters they would like to have if thdy'won'a raffle. 
School groups who had been randomly selected to see the astronomy 
•exhibit were compared to groups who had not seen the exhibit, 
(before the exhibit was open to the public). Of the visitors 
who had seen the exhibit, chose the. astronomy poster, versus 
461 of the control group. , ' 

* V 

41 A fourth kind of ^evidence comes *fjom visitor answers to multiple v.- 
choice questions about astronomer 

• • 

This has a major drawback: it's likely to-show that visitors 
haven't learned anything because: 



whether an exhibit 



a) 
b) 

« 

c) 



it's difficult" to design good questions. 



/ 



the mode of learning was participatory, and the mode 
of responding is pencil and paper. ^ , 

visitors really might not have learned anything about ' 
the questions, but might rush out and buy six books on* 
astronomy or tell ten of their friends: "You've got to , 
see the astronomy exhibit at LHS". * t 

Bat everyone always wants to ask multiple choice questions , so if you 
can't resist, here are some guidelines: 

a) ..Visitors rarely/ are willing to answer mote than four questions, 
?o after their ag'e'and sex'you are left with two questions. 



b) 



Two questions" are fine, just don't give every visitor the 
same question. You can have .a group of 20 questions and 
make 10 different-questionnaires. 



v 



5) 



c) It's not necessary to ask more than. 30 subjects from one 
category the same question. If this does not yield at ' 
least a trend in favor of your program, change the questions, 

* exhibit, or something. - ^ 

d) ' To compare visitors who have seen an exhibit with those who v 
„ have not, give half of them the questions before they view 

and half after they view.. c . 

A fiftl^and very important : kind of evidence is a participatory 
intervrew. For the astronomy exhibit, we might ask visitors tq \ 
"Focus the telescope oh Saturn", and determine how long it takes 
them to succeed. This is better than a multiple choic^question 
because it's in the same mode as the exhibit. Rather/than trans- 
ferring informs ti on *£sem a 'participatory mode to a. written mode, 
the skills- presented in the exhibit are assessed. 
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In assessing ^ubit effectiveness, it is our^ecommendat ion that as 
many different kinds of evidence as possible be gathered. Even if each bi# 
of evidence .is questionable in some way, if the-program is. effective, the 
total evaluation picture will describe it. 

Inevitably^ w^in this approach is taken some Information is inconsistent 
with other information. This Is also of interest*. Sometimes ,a\ tempts to 
reconcile such inconsistencies result in great increases in understanding. 
Sometimes one is left with reporting the inconsistencies and. waiting for 
more information^ For instance, a study at LHS indicated that some children 
lov$d an exhibit while others found it boring. More detailed analysis 



indicate that length of exposure to tne exhibit, age, previous science courses 
or some other variable explains this finding. * 

Providing the Informed Decision Making model fof* exhibit developers 
at LHS/ has had an important effect: *hey have learned to do formative 

evaluation. One reason that evaluation has enjoyed such a positive - ' 

" ' ' * / ■ ' ' ' • 

reception is clearly that it has. worked: exhibits have bee^improved sub- 
stantially A?Sthe result of formative trials. - Also, we have learned alot 
about who our users are- and'. how to attract more of them. Finally the " 

director has vociferously supported formative evaluation. Administrative 

, / • * ' • - 

support is essential for implementation of formative evaluation techniques 

Staff . members attuned to forma tt/e evaluation le"arn to: 1) value ^sitor 

particular 



exhibit two different ways to see which is best, and 3) look for choices 
rather than answers. . 



/ 



COSTS OF INFORMED DECISIO Nj^ING „ 

It is, of course, difficult to assign ah actual cost to an improved 
exhibit. Nevertheless, tlje quality of an exliibit is important to draw and 
keep visitors. How do we know that formative evaluation is cost effective? 



P»8« » 

One way^to' assess this is to Look at the costs of the evaluation; When 
an in-house evaluator in conjunction with the, project staff carries out 
the evaluation, the actual added cost is the evaluatoi\'s -salary. Staff 
time spent in evaluation is also a cost of the evaluations But an 
efficient evaluation might actually save staff -time by reliably and 
objectively answering critical questions. Hiring outside eval^tbn teams 
is usually expensive and may not result in as much staff participation and * 
staff learning as having a staff evaluator. It should be noted thaV outside 

... . .....'».. : t .. 

evaluators have a very definite and worthwhile role as unbiased observers 
and ^re often considered to be neces|ary for summative studies. 



CONCLUSIONS 



V 



Formative evaluation is a valuable tool when used effectively by 

exhibit development teams. The Informed Decision Making model has proved 

<r - 

effective as a formative evaluation technique; In this approach the 

evaluator first creates choices and then gathers evidence to determine the 

« 

best choice. By creating choices in conjunction with the project staff, the 
evaluator is .sure to. work on. problems of interest to the museum. By gathering 
evidence from observinjg actual museum visitors, the evaluator can help make 
the museum programs more Relevant and' comprehensible to the audience. 

Although the problem is complicated, as our understanding of the 
museum learning environment grows, our ability to/ design better exhibits will 
also increase. Wfrife summative evaluation questions about .the learning 

► ft 

potential in a museum are iimportant, it is clear that learning is more likely 
,to take place when exhibits^ are designed to suit the users. Museums succeed 
when visitors return -- visitors return when* they enjoy their experiences. 
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FIGURE;- CAPTIONS: 
1. 



.Family groups often interact with exhibits - one member 
may adopt a teaching role. . 



2. Visitors come in many sizes. 
3.. The 1 LHS computer exhibit is" reiy popular. 

4. This exhibit does not interest the whole visitor group. 
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